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Abstract 

In the 1980s, the National Aeronautics and Space 
Administration (NASA) embarked upon a major Earth- 
focused program called Mission to Planet Earth . The 
Goddard Space Flight Center (GSFC) was selected to 
manage and develop a key component— the Earth 
Observing System (EOS). The EOS consisted of four 
major missions designed to monitor the Earth. The 
missions included 4 spacecraft: Terra (launched 

December 1999), Aqua (launched May 2002), ICESat 
(Ice, Cloud, and Land Elevation Satellite, launched 
January 2003), and Aura (scheduled for launch January 
2004). The purpose of these missions was to provide 
support for NASA ’s long-term research effort for 
determining how human-induced and natural changes 
affect our global environment . 

The EOS Data and Information System (EOSDIS), a 
globally distributed, large-scale scientific system, was 
built to support EOS. Its primary function is to capture, 
collect, process, and distribute the most voluminous set of 
remotely sensed scientific data to date estimated to be 350 
Gbytes per day. The EOSDIS is composed of a diverse 
set of elements with functional capabilities that require 
the implementation of a complex set of computers, high- 
speed networks, mission-unique equipment, and 
associated Information Technology (IT) software along 
with mission-specific software. 

All missions are constrained by schedule, budget, and 
staffing resources, and rigorous testing has been shown to 
be critical to the success of each mission. This paper 
addresses the challenges associated with the planning, 
test definition, resource scheduling, execution, and 
discrepancy reporting involved in the mission readiness 
testing of a ground system on the scale of EOSDIS. The 
size and complexity of the mission systems supporting the 
Aqua flight operations, for example, combined with the 
limited resources available, prompted the project to 
challenge the prevailing testing culture. The resulting 
success of the Aqua Mission Readiness Testing (MRT) 
program was due in no small measure to re-structuring 
the traditional programmatic and technical approach to a 
more efficient and robust program. Programmatically, it 
meant gaining the endorsement, commitment, and 


cooperation of the numerous subsystem element 
managers and other stakeholder organizations. 
Technically, it required an MRT program that was agile, 
could rapidly adapt to requirements changes, and was 
flexible in its overall approach. Furthermore, this paper 
addresses the following questions: 

1. What are the key “ingredients ” (e.g., test tools, 
organization) needed to conduct a successful 
MRT program? 

2. What distinguishes EOS MRT from the 
traditional system testing approach? 

3. Where should the focus of testing be since it is 
infeasible to test every element or subsystem? 

4. How can MRT be applied effectively to other 
systems or missions? 

To provide answers to these questions, this paper 
relies heavily on real-life, hands-on experiences (“ lessons 
learned”) gained during mission readiness testing of the 
Terra ground system and, most recently, the Aqua and 
ICESat missions. Moreover, this paper explores how 
lessons learned were turned into “ lessons applied” for 
the upcoming Aura mission. Although derived from the 
EOS missions, MRT techniques and strategies can be 
applied to enhance the testing of other missions. 

1. Introduction 

/ 

With the recent successes of the Terra, Aqua, and 
ICESat missions (and Aura scheduled for launch in early 
2004), NASA’s Earth Science Enterprise will soon be 
realizing its vision of having a fleet of Earth-observing 
platforms continuously “taking the pulse” of the Earth and 
its environs in an effort to improve prediction of weather, 
climate, and natural hazards using the unique vantage 
point of space. To accomplish this feat, the Earth 
Observing System (EOS) was conceived in the 1980’s; the 
Goddard Space Flight Center was selected to design, 
implement, manage, and operate the EOS over its lifetime. 
Over the past ten years, the EOS has evolved and matured 
in phases and was finally commissioned in December 
1999 with the successful launch of the Terra spacecraft. 



The EOS linchpin, the EOS Data and Information 
System (EOSDIS) was envisioned to support the " 
estimated 350 billion bytes per day of Level-0 science 
data that will be generated from the four EOS spacecraft. 
The EOSDIS consists of a diverse set of integrated 
elements with functional capabilities that required the 
implementation of a complex set of computers, high-speed 
networks, mission-unique equipment, and associated 
Information Technology software. EOSDIS capabilities 
include the processing of all data flows for the four EOS 
missions, from the flight operations control center that 
manages the health and safety of the spacecraft, to the 
extensive distributed systems that provide science data 
processing, archive, and distribution services. The 
EOSDIS is an integrated part of the EOS Ground System 
(EGS), which includes NASA institutional capabilities 
such as the Space and Ground Networks for satellite 
communications, the EOS Mission Support and Science 
Support networks, and the Flight Dynamics Facility. 
Complete and effective testing of this large and distributed 
ground system is vital to the success of the EOS missions. 

In order to ensure mission success, EOSDIS management 
chartered a mission readiness test team under the direction 
of a Mission Readiness Manager (MRM). The ground 
system supporting each of the four missions would be 
thoroughly tested from a mission operations perspective 
along with all of its associated Ground Network (GN) and 
Space Network (SN) assets. 

The initial concept for a comprehensive EOS mission 
readiness test program, conducted by an independent test 


team, was applied to the first mission, Terra. However, 
because of significant problems with the original EOS 
Operations Center (EOC) architecture (e.g., real-time 
performance, software delays), along with associated 
launch schedule pressures, the original plan was 
abandoned, and all mission readiness testing was 
conducted in conjunction with the Flight Operations Team 
(FOT). Although solving the immediate schedule 
pressures, the overall testing effort did not achieve the 
desired confidence in ground system performance. It was 
concluded that the model previously used on small 
Goddard missions, in which the flight operations team 
controlled the ground system test program, was not 
adequate for large missions like Terra and Aqua, which 
required a large FOT, complex mission scheduling, and 
ingest of large volumes of science data. For the upcoming 
Aqua mission, senior management challenged the Aqua 
MRM to develop a more effective mission readiness 
testing model. After a thorough review of how testing was 
conducted for the Terra mission (see “Lessons Learned 
from the Terra Testing Experience”), the decision was 
made to “re-architect” the overall Mission Readiness Test 
(MRT) program. Refer to Table 1 for the major features 
incorporated into the revised model. Employing a 
combination of best system engineering practices and 
program management skills, we successfully implemented 
this new model for the remaining EOS missions. The 
remainder of this paper will discuss how this new model 
enabled us to test successfully NASA’s largest ground 
system. 



Figure 1 EOS Ground System 






2. Lessons Learned from the Terra Testing 
Experience 

The Terra mission was the first in the planned series of 
EOS spacecraft. The spacecraft and ground system 
elements were new and complex, requiring extensive 
development resources. Numerous development problems 
for the many first-generation systems had to be resolved, 
and the test engineers had to learn about the architecture 
and planned capabilities of the ground system as they 
evolved. 

The requirements against which the mission readiness 
testing was conducted were not originally developed to be 
a requirement set; rather, they were a compilation of 
functionality to capture the current ground system design. 
Due to time constraints with launch scheduled less than 7 
months away (launch occurred 12 months later), no 
requirement analysis or improvement was performed. As 
a result, the test team labored to test and verify them as 
written. For example, individual requirements often 
specified multiple functions and were very difficult to 
provide a status within one test event. Many requirements 
ended up with ‘Partial Pass’ statuses that were never 
completed. However, the set of requirements was useful 
in that it did derive from valid sources (e.g., ground 
system documents) and did address a significant portion 
of the capabilities of each element providing Terra flight 
operations support. 

Unfortunately, other levels of testing were also 
affected. Element system development often suffered 
from poor configuration management practices and some 
inconsistent interpretations of ambiguous requirements. 
The EOC systems were redesigned radically, but the 
requirements governing their functions were not revised 
completely. All ground element participants became part 
of ‘the team’ supporting flight operations team tests. With 
the Terra flight operations team leading all the tests, the 
MRM’s role became largely that of a test witness, as 
opposed to a test director. The lack of direct oversight of 
end-to-end data flows increased the potential that system 
problems could go undocumented. 

The MRM maintained a paper system for recording 
verification statements, which were signed by element 
representatives. In some cases, previous element 
Acceptance Test results were cited as evidence of 
requirement verification, rather than independent test 
results. Overall, the critical evaluation and the criteria for 
assignment of statuses were less rigorous than those 
applied for later missions. 

As the first EOS mission, it was expected that Terra 
ground system testing would involve growing pains. The 
excessive number of problems experienced was noted by 
many, and recommendations were bundled into a variety 


of lessons-leamed packages, to be applied to Aqua testing. 
Some of these were: 

1. Spacecraft and ground system organizations 

need to have clearly defined management 
chains, with individual roles and responsibilities 
understood within and outside of each 
organization. Communication between elements 
needs to be improved. 

2. Requirements need to be unambiguous, testable, 

and consistent. Improved requirements 
management and maintenance are needed. 

3. Configuration management of systems, 

schedules, and test data files is important and 
should not be done lightly. Configuration 

changes and data updates need to be 
communicated to all affected elements as soon 
as possible. 

4. Realistic schedules are needed, so that plans 

based on them are believable and usable. Plans 
need to reflect system/functional availability 

dates. 

5. Additional complex, end-to-end test scenarios 
are needed, using realistic operating procedures 
and high-fidelity data sources. A single test 
director is needed for such tests to ensure that 
resources and schedules across test groups are 
consistent, reasonable, and do not conflict. 

To implement these lessons learned, a new testing 
approach would be needed for the upcoming Aqua 
mission. The MRM made the decision to examine a more 
systematic and rigorous technical approach and to adopt a 
different test management approach as well. 

3. Uniting the Forces of Systems Engineering 
and Project Management 

The MRM decided to assess the applicability of a 
systems engineering methodology, similar to that 
successfully used for developing spacecraft hardware and 
associated systems, for a mission readiness testing 
program. From this review, it was concluded that a 
systems engineering approach would provide the 
necessary framework and foundation for our revised 
mission readiness testing plan, and thus we incorporated 
many such principles into our testing program for Aqua. 
Moreover, we reviewed the ideas behind successful 
project management, and by melding the best practices of 
systems engineering and project management, we were 
able to define a new model for the Aqua mission. 



Table 1. Salient Features of New Model 



The International Council on Systems Engineering 
(INCOSE) defines systems engineering as “an 
interdisciplinary approach and means to enable the 
realization of successful systems. It focuses on defining 
customer needs and the required functionality early in the 
development cycle, documenting requirements, then 
proceeding with design synthesis and system validation 
while considering the complete problem.” Furthermore, 
“systems engineering integrates all the disciplines and 
specialty groups into a team effort forming a structured 
development process that proceeds from concept to 
production to operation. Systems engineering considers 
both the business and the technical needs of all customers 
with the goal of providing a quality product that meets the 
user needs.” 

We began our technical activities by focusing on 
specific Aqua functionality and interfaces that would drive 
our test requirements, with an eye on the end-to-end 
operational system. Early on, we worked closely with the 
various ground system element managers and the Aqua 
FOT to develop a set of mission readiness requirements 
and to ensure that they understood our test goals and 
objectives as well as the need to become active 
participants and team members in our overall testing 
program. Finally, to ensure that this approach would 
work, we enlisted senior management’s support for this 
alternate model. This enabled us to establish and maintain 
a strong measure of independence while developing, 
executing, and documenting this alternate mission 
readiness testing model. The next few sections describe in 
more detail the results of our implementing this alternate 
testing model for the Aqua and ICESat: missions. 


4. Implementing the New Mission Readiness 
Test Model for Aqua 

The shortfalls of the Terra mission readiness test model 
influenced heavily our decision to have a more active 
Mission Readiness Test Team (MRTT) be responsible for 
testing the Aqua ground system. In addition, we needed a 
team that could take a more proactive, independent 
approach. This section details how we applied key 
principles from both the systems engineering and project 
management disciplines to the Aqua test model, resulting 
in the successful Aqua test program (and beyond). 

4.1 Defining and Documenting Requirements 

Important aspects of successful systems engineering for 
complex systems include understanding objectives and 
conducting rigorous requirements identification, analysis, 
and management. In an ideal world, requirements flow 
down from an operations concept document, to ground 
system, spacecraft, and instrument specification 
documents [1]. The MRTT would then use the ground 
system document to develop the specific test requirements 
for verifying the readiness of the ground system to support 
launch and early-orbit operations, thus enabling complete 
traceability forwards and backwards. Unfortunately, good 
requirements traceability and testability did not exist for 
the Aqua mission due to lack of rigor in the development 
of the Aqua ground system document. 

Convinced of the importance of having a solid set of 
test requirements, the Aqua MRTT expended a significant 
amount of time and effort on the requirements analysis 
phase. We drafted an initial set of Mission Readiness 
Test Requirements (MRTRs) based upon a thorough 
review of the Project documentation that was available at 
the time, including the Mission Operations Concept 
Document, Ground System Requirements Document, and 
various Interface Control Documents. The MRTRs were 
then documented in the Aqua Mission Readiness Test 
Plan (MRTP) [2] and widely circulated via email and web 
access [3] for analysis and updates from the ground 
system element managers. 

A systems engineering focus on missions operations 
emphasized the importance of concentrating on 
performance requirements as well as basic functionality. 
As additional satellites are supported by the EOS ground 
system, the elements’ and networks’ abilities to sustain the 
increasing data processing loads must be proven prior to 
each launch. For Aqua, it was determined that single 
mission tests would not be adequate for all interfaces, so 
performance MRTRs were developed and multi-mission 
data loading tests were considered where feasible. In 
addition, error checking and high- volume data processing 
stress tests were known to be valuable, but the creation of 


special test data configurations and test data files requires 
a great deal of time and effort. These specialized tests can 
expose problems that would otherwise not surface until 
late in the testing or even during operations. On Terra, 
inadvertent data mistakes and inadvertent operator 
manipulations generated more error conditions than any 
set of test cases designed to test error handling capabilities 
of the systems under test. 

Once we had a draft set of MRTRs specifying all the 
functions that were within the scope of our testing 
program, we conducted two analyses that helped to clarify 
and finalize the MRTR set. The first was a simple pairing 
of “send” and “receive” requirements. For each MRTR 
specifying that Element A send a data item to Element B, 
there would be a separate, consistent MRTR for Element 
B to receive that data item from Element A. The reason 
for having two requirements for one interaction is that the 
test team needed to be able to assess system statuses 
independently, since often only one side of an interface 
fails. 

Next, to address issues of end-to-end mission 
perspective, we conducted a “thread analysis” of the 
mission data flows. Data flow threads represented the end- 
to-end flow of a single data item through the nodes of the 
integrated ground system. Each data item was traced from 
an original sender through one or more intermediate nodes 
until it either reached the final receiver or was changed 
significantly into a different data item (at which point a 
new thread was started). Due to the large number of Aqua 
requirements and the complexity of the EGS, the Thread 
Analysis proved to be rather time consuming; however, 
several previously undocumented requirements were 
uncovered which allowed early resolution of system 
design shortcomings. 

Another area of systems engineering that we applied 
was the identification of high risk areas as early as 
possible in the program. During our requirements 
discussions with the element managers, we paid particular 
attention to those requirements that involved new or 
complex interface implementations, implying a high or 
significant risk from a testing and operational standpoint. 
These risks were documented and forwarded to the 
Project for on-going risk mitigation. 

The process to develop a comprehensive, testable, and 
accurate set of Mission Readiness Test Requirements was 
a time-consuming effort. Requirements traceability was 
not as straight forward as was hoped, so to compensate, 
the team maintained close coordination and 
communication with each Element of the ground system 
throughout the development process. This enabled the 
team to continuously update the MRTRs to ensure they 
remained an accurate up-to-date reflection of the 
operational system. As launch readiness reviews were 
conducted, it became apparent that several dozen MRTRs 


did not have traceability to upper level documents. The 
missing traceability was, in fact, due to functionality 
missing in the higher level documents. The MRTT’s 
methodical systems engineering approach used all levels 
of documents to develop its comprehensive set of 
requirements, especially the Interface Control Documents, 
operational agreements, and lower-level requirements 
specifications. The experience and vast systems 
knowledge of each member of the test team were key to 
verifying the Aqua ground system for launch. 

4.2 Exercising Good Project Management Skills 

The most important skills that were injected into the 
Aqua model included the following: 

• Strong leadership throughout the test team 

• Team-building principles to develop a high- 
performance test team 

• Inclusive, “open” environment 

• Frequent communications 

Based upon the comments from the post-Terra review, 
we made a concerted effort to improve our working 
relationships with all ground system elements and to foster 
an environment of teamwork. To that end, we 
concentrated on improving overall communications, both 
written and oral, among all element managers and the 
FOT. Our first initiative was to get early concurrence on 
MRTR definitions, test constraints, and detailed success 
criteria from the element managers. We conducted in- 
person, early reviews with element representatives to 
define, finalize, and agree on all mission readiness test 
requirements, in order to permit testing to start as soon as 
various capabilities became available. Early element 
concurrence on the MRTRs was essential to support a 
common understanding of what was expected (for 
functionality and performance) and how requirement 
success or failure would be evaluated. 

To foster an inclusive environment, we held bi-weekly 
MRTT working group meetings with all test team 
members to discuss topics related to upcoming tests. 
These MRTT working group meetings were held early in 
the ground system development cycle, to support test 
development and to identify potential issues early in the 
test planning cycle. This was important to the MRM who 
was now in a better position to identify elements that were 
behind in their development deliveries and to potentially 
gain insight into data paths that could not be exercised 
until very late in the test schedule. At these meetings, we 
strived to resolve any issues quickly and satisfactorily, 
working very closely with the element managers. 

Our desire to establish an independent posture 
throughout the testing program was mentioned previously. 



An independent perspective was necessary to ensure the 
integrity of the test data, system processing, ‘operational’ 
approach, and test results. Furthermore, we felt strongly 
that an independent, “big-picture” perspective would keep 
the test team focused on performing tests correctly, not 
quickly, and conducting tests under the most realistic, 
operational conditions possible. Also, this independence 
allowed us to look for any “gaps” in the overall Aqua end- 
to-end test program conducted not only by us but also by 
other groups, the FOT for example. Key to this 
independence was maintaining an unbiased assessment of 
the current requirement status, which was vital to 
maintaining trust in the test reporting and verification 
process. Although challenged at times to lessen our 
independence, we relied on strong leadership from within 
the test team to resist these attempts. 

4.3 Verifying the Ground System Functionality 

Once the MRTRs were well defined and documented, 
test scenarios that contained progressively more complex 
ground system operations were defined and reviewed by 
the element managers. After a thorough review, the test 
scenarios were also documented in the MRTP [2] and 
served as the basis for developing the actual mission 
readiness tests to be executed. Key of course to a 
successful test program was the actual execution of the 
scenarios to verify the functionality and performance of 
the ground system in question. The following describes 
some observations about our testing approach that 
ultimately had a strong bearing on why we were so 
successful. First of all, we developed a suite of tests that 
at a minimum provided coverage for all of the launch 
critical requirements. Our test program consisted of 
twelve tests that ran the gamut of simple “point A to point 
B” data flow tests to a full-blown, integrated systems test 
incorporating all of the ground system elements that 
would be necessary to operate the Aqua mission (see 
Figure 1 for a graphical depiction of this test). 

Second, as our test schedule allowed, we tried to “test 
early, and test often.” However, any decision to run the 
test ‘early’ depended upon the specific capabilities 
missing, how soon they were expected to be available, and 
the benefits to be gained from a partial test run. Running 
a test too early could lead to a non-productive use of 
element resources and running a test too late might not 
allow a sufficient recovery time should serious problems 
be uncovered. However, before we began any serious 
mission readiness testing, we met with the element 
managers to ensure that system-level testing and 
acceptance testing had been conducted for their particular 
element; otherwise, the tests might fall short in attempting 
to verify functionality. For example, during our testing of 
the Ground Network sites, some of our Aqua MRTs 


turned into unplanned engineering/system-level tests, 
involving more system and human resources than 
necessary. All interface testing should be completed prior 
to MRTs to verify the data product and data transfer 
conformance to the Interface Control Documents (ICDs). 

Another important success factor was the early 
definition and availability of good test data. Telemetry 
and data handling tests, especially those involving the 
Space or Ground Networks, are only as good as the test 
data being used. We strived to obtain realistic data early, 
to validate and document the contents, and to deliver files 
to each element for test use. Actual spacecraft data sets 
were essential for testing some elements of the ground 
system under operational conditions. Early spacecraft 
data had limitations. With each database change by the 
spacecraft or instrument developers, the data became out 
of date and less useful for detecting data path anomalies, 
because of format or mnemonic limit changes in the 
database. Early database updates tended to be more 
extensive than those close to launch. Real data was 
obtained from the spacecraft vendor as soon as possible 
after database updates to help minimize such problems. 

Demonstrating repeatability was critical to validating 
the ability of the ground system to support mission 
operations. We also attempted to ensure sufficient time in 
the test schedule for late regression testing. This became 
particularly important around the ground system ‘freeze’ 
date. Probably the only time testing can occur on the real, 
to-be-flown ground system configuration is after Ground 
System ’’freeze”, so we planned on having extra time in 
our schedule for last-minute tests. 

We never assumed that any hardware or software 
change was “transparent.” Software and hardware 

changes almost always affect previously-tested 

functionality, so we insisted that all changes to ground 
system elements’ hardware, software, and configurations 
be reported to the MRM and closely tracked and assessed 
by the MRTT for potential impact to previous, current, 
and future testing. It was especially important to have a 
strong process in place to inform the MRTT of all changes 
to key ground network and space network configurations 
and other NASA institutional capabilities. Failing that, 
we would have been back in the Terra environment where 
changes were made without the proper procedures in- 
place to track them. 

Lastly, an important process that we adhered to was the 
documentation of Discrepancy Reports (DRs) against all 
anomalies, not just major ones. We assigned an MRTT 
member to write each DR found during testing: too often a 
problem written by the developer did not include the 
specific problem found by MRTT and did not specify that 
the problem was found during an MRT. It was also 
important to retain artifacts to support the discrepancy 
analysis. In addition, we maintained an MRTT 



independent presence on the Review Board to ensure that 
MRTT-generated DRs received proper attention and 
disposition. Finally, we made sure that adequate testing 
was done prior to closure of each DR. It should be noted 
that some conflicts did arise when an element assigned a 
status to a requirement that did not agree with the 
observations of the MRM or the other test engineers. 
Since the elements had been given the responsibility to 
status their own requirements and maintain their own 
artifacts, these situations were sometimes difficult to 
resolve and resulted in a requirement status of ‘Open 5 
being recorded while the issue was investigated further. 
But, having an open, cohesive team environment in-place 
significantly reduced any major disagreements between 
the MRTT and element manager. 

5. Application of the Aqua Mission Readiness 
Test Model to the ICESat Mission 

ICESat (the Ice, Cloud, and Land Elevation Satellite) 
was another EOS mission that used many of the same 
components as Aqua, but also gave responsibility for large 
parts of ground system to the spacecraft developer. The 
ICESat mission was a smaller mission (single instrument) 
with a correspondingly smaller test team. Several EOS 
Ground System components that support Terra and Aqua 
also provided ICESat support (spacecraft 
communications, data transport and processing, etc.). The 
main complicating factor was that the ICESat control 
center and flight operations team were located remotely 
from GSFC, as part of the Laboratory for Atmospheric 
and Space Physics (LASP) University of Colorado, and 
were operated under a contract to Ball Aerospace. 

Ground systems test preparations for ICESat were 
nearly concurrent with Aqua test efforts, so the test 
planning and test management principles previously 
discussed that were applied to improve the Aqua mission 
readiness test effort were also applied to the ICESat effort. 
The mission readiness test team operated as an 
independent entity, monitoring the progress of ground 
system elements and reporting directly to the project on 
testing progress. MRTT meetings tended to cover a broad 
range of topics, functioning as more of a testing and 
operations working group and fostering a regular and 
beneficial exchange of information among all ground 
element representatives. 

Mission Readiness Test Requirements (MRTRs) were 
developed from the ICESat Mission Operations 
Requirements Document and from knowledge of the 
participating ground system capabilities. The MRTP [4] 
was developed with test scenarios similar to those 
conducted for Terra and Aqua, but with modifications for 
different instrument and science data flow interfaces. 


ICESat mission readiness testing was characterized by 
both hands-on testing of ground system elements and 
“outsourced” testing. The MRTT performed active 
testing of the ground system elements and interfaces, and 
the Mission Readiness Test Engineers occasionally 
observed in a passive role when LASP conducted internal 
readiness tests and simulations. During multi-mission 
tests with other, higher-visibility EOS missions, ICESat 
test objectives were not the drivers of the test scenarios 
but were fully included. Close cooperation and 
persistence were required to ensure that this happened. 

Tests were conducted as early as feasible. Early 
science system tests were run as partial test cases, using 
the early versions of science data processing systems, as a 
way for the MRM to get an early indication of element 
readiness and for the system developer to ‘acceptance test’ 
their system in a semi-operational environment. . These 
tests helped identify basic interface and system problems 
while it was still easy to implement fixes. 

Sometimes, testing results surprised the participating 
elements - ICESat data rates and complexity seemed low 
relative to other larger missions, yet almost every test 
found something that needed to be fixed, and in a few 
cases called for new code. In all aspects, the principles 
that were developed for Aqua testing and applied to 
ICESat were proven to be effective and useful. The 
successful application of these principles to the ICESat 
mission also demonstrated that the guidelines and 
processes necessary for testing very complex missions 
(e.g.. Aqua) are scalable and are important for complete 
testing of smaller and less complex missions such as 
ICESat. 

6. Challenges Remain for Aura Mission 
Readiness Testing 

Despite the successes achieved during the mission 
readiness testing for the Aqua and ICESat missions, 
several challenges remain for the upcoming Aura mission 
(scheduled for an early 2004 launch). Although the 
spacecraft and ground system are very similar to that used 
for Aqua, the requirement remains to fully test the ground 
system within a multi-mission environment. Also there is 
a strong desire to test the overall performance during data 
“outages” and hence test the resiliency of the ground 
system to recover to nominal data processing operations. 

As mentioned in the Introduction, the EGS will be 
generating 350 billion bytes of Level-0 science data per 
day from all four satellites. The challenge for the Mission 
Readiness Test Team is to develop and execute a series of 
anomaly tests (within an existing operational 
environment!) that will generate a series of detailed 
metrics that can be used to assess how the EGS will react 
to a failure of one or multiples of the ground system 


elements. The results of these tests could be useful in 
determining where the “chokepoints” are and where the 
insertion of additional backup capabilities might be 
needed. 

Even though Aqua MRT was performed in an 
operational environment, there was only one spacecraft 
on-orbit (Terra). Now that Aqua and ICESat have joined 
the science data gathering community, there are 
significant time and resource restrictions imposed on 
testing. For instance, during Aqua, MRT could request a 
Ground Station support of EOS resources for a 4 hour 
block of time. Now, Aura must utilize EOS Ground 
Station resources in 1 hour blocks sandwiched between 
Aqua and ICESat Science data dumps. Additional test 
planning and coordination is required to prevent any 
potential conflicts between the test effort and on-orbit 
operations. 

Another challenge is to perform a series of regression 
tests on the telemetry and command software that will be 
delivered to support the Aura mission. For the Aqua 
mission, considerable effort was expended in testing the 
major functionality of the software, and the test team 
gained significant experience in this area. However, in the 
future, they will not only have to test the new software 
required for the Aura mission, but will have to ensure that 
existing functionality is not compromised with the latest 
Aura release. This will require an in-depth knowledge of 
the telemetry and command software and how the existing 
system operates today. 

Programmatic challenges remain for Aura MRTT. 
Although budgets always impose constraints on projects, 
and are particularly tight on testing activities, Aura, being 
the last of the three planned EOS missions has an 
especially tight budget. The Mission Readiness Test 
Team has, and must continue to, look at ways to 
streamline the tests while providing the necessary level of 
completeness and attention to detail to ensure another 
successful test campaign. As Aura Project Management 
decisions influence the scope and breadth of testing, it is 
the MRTT’s responsibility to identify, evaluate, and 
inform management of potential risk areas. 

Many EOS contracts phase down as Aura launch 
approaches and then terminate or transition to new 
contracts after launch. With the current tight job market 
persons are anxious to ensure their job security. This may 
lead to experienced members leaving for other 
opportunities. Steps need to be implemented before 
individuals ‘move on’ in order to maintain the system 
knowledge base and the processes/procedures that enabled 
the MRTT to be so effective on Aqua. 

In addition to the attrition of ‘hands-on’ persons, 
changes in management are occurring within contractors 
and at the GSFC. New management brings new 
experiences and concepts of how to conduct effective 


testing. Providing new management with lessons learned 
[5], and establishing regular meetings to communicate 
current status and discuss previous experiences will help 
to ensure a smoother transition. 

7. Concluding Remarks 

The revamped mission readiness test program for Aqua 
was highly successful. All major functional and 
performance criteria were met, and only a small number of 
ground system anomalies were reported (compared to 
Terra!) during on-orbit operations, and those that did 
occur were fixed very quickly. In our opinion, the rock- 
solid performance of the EGS to date has validated our 
mission readiness testing model and our approach for 
implementing it. Several conclusions can be posed at this 
time: (1) the systems engineering methodology can be 
successfully adapted to a mission readiness test program; 

(2) adhering to core project management guidelines 
created the necessary environment in establishing a 
cohesive test team; (3) investing the effort and schedule 
into developing a strong requirements base was rewarded; 
(4) this new testing model proved to be scalable (for 
example, ICESat); and (5) maintaining an independent 
posture while at the same time fostering teamwork created 
the right environment for testing a ground system as large 
and complex as the Aqua ground system. 
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